Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OpenSearch crashes on closed client connection before search reply when total ops higher compared to expected #4143

Merged
merged 1 commit into from
Aug 5, 2022
Merged

OpenSearch crashes on closed client connection before search reply when total ops higher compared to expected #4143

merged 1 commit into from
Aug 5, 2022
+61 −1

Conversation

reta
Copy link
Collaborator

@reta reta commented Aug 5, 2022

Signed-off-by: Andriy Redko andriy.redko@aiven.io

Description

In some circumstances, when running a search operation, if the client TCP connection is closed before the search operation is completed, opensearch crashes with:

[2022-07-20T17:51:36,145][ERROR][org.opensearch.bootstrap.OpenSearchUncaughtExceptionHandler] fatal error in thread [opensearch[mycluster-elasticsearch-3][search][T#8]], exiting
java.lang.AssertionError: unexpected higher total ops [15] compared to expected [14]
	at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:465) ~[opensearch-1.3.4.jar:1.3.4]
	at org.opensearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$2(AbstractSearchAsyncAction.java:303) ~[opensearch-1.3.4.jar:1.3.4]

Issues Resolved

Closes #3962

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@reta
OpenSearch crashes on closed client connection before search reply wh…
addd0a8 
…en total ops higher compared to expected

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
@reta reta requested a review from a team as a code owner August 5, 2022 17:21
@github-actions
Copy link
Contributor

github-actions bot commented Aug 5, 2022

Gradle Check (Jenkins) Run Completed with:

@codecov-commenter
Copy link

codecov-commenter commented Aug 5, 2022 •
edited by codecov bot
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.64%. Comparing base (fbe93d4) to head (addd0a8).
Report is 2980 commits behind head on main.

Additional details and impacted files 
@@             Coverage Diff              @@
##               main    #4143      +/-   ##
============================================
+ Coverage     70.55%   70.64%   +0.08%     
- Complexity    57013    57093      +80     
============================================
  Files          4603     4603              
  Lines        274478   274482       +4     
  Branches      40201    40201              
============================================
+ Hits         193658   193905     +247     
+ Misses        64511    64336     -175     
+ Partials      16309    16241      -68

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@andrross andrross merged commit 203f44e into opensearch-project:main Aug 5, 2022
@andrross andrross added the backport 2.x Backport to 2.x branch label Aug 5, 2022
opensearch-trigger-bot bot pushed a commit that referenced this pull request Aug 5, 2022
@reta @github-actions
OpenSearch crashes on closed client connection before search reply wh…
10b37fd 
…en total ops higher compared to expected (#4143)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
(cherry picked from commit 203f44e)
@opensearch-trigger-bot opensearch-trigger-bot bot mentioned this pull request Aug 5, 2022
Merged
@dbonf
Copy link

dbonf commented Aug 5, 2022 •
edited
Loading

will this be backported to 1.3 please?

@reta
Copy link
Collaborator Author

reta commented Aug 5, 2022

will this be backported to 1.3 please?

Yes, on it

reta added a commit to reta/OpenSearch that referenced this pull request Aug 5, 2022
@reta
OpenSearch crashes on closed client connection before search reply wh…
2a905cd 
…en total ops higher compared to expected (opensearch-project#4143)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
@reta reta mentioned this pull request Aug 5, 2022
Merged
reta added a commit to reta/OpenSearch that referenced this pull request Aug 5, 2022
@reta
OpenSearch crashes on closed client connection before search reply wh…
af4e9dd 
…en total ops higher compared to expected (opensearch-project#4143)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
@reta reta mentioned this pull request Aug 5, 2022
Merged
@reta reta mentioned this pull request Aug 5, 2022
Merged
@reta reta mentioned this pull request Aug 5, 2022
Merged
reta added a commit to reta/OpenSearch that referenced this pull request Aug 5, 2022
@reta
OpenSearch crashes on closed client connection before search reply wh…
20db089 
…en total ops higher compared to expected (opensearch-project#4143)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
@reta reta mentioned this pull request Aug 5, 2022
Merged
@reta
Copy link
Collaborator Author

reta commented Aug 5, 2022

@dbonf would you be able to give it a try on 1.3.5-SNAPSHOT? thank you!

@dbonf
Copy link

dbonf commented Aug 5, 2022

@reta sure, do I need to compile it or is it already available somewhere?

reta added a commit to reta/OpenSearch that referenced this pull request Aug 5, 2022
@reta
OpenSearch crashes on closed client connection before search reply wh…
972a776 
…en total ops higher compared to expected (opensearch-project#4143)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
reta added a commit to reta/OpenSearch that referenced this pull request Aug 5, 2022
@reta
OpenSearch crashes on closed client connection before search reply wh…
a2e79af 
…en total ops higher compared to expected (opensearch-project#4143)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
@reta
Copy link
Collaborator Author

reta commented Aug 5, 2022

@reta sure, do I need to compile it or is it already available somewhere?

Good question, I think if you could compile (once #4148 is in), would be awesome, I am not 100% sure we publish snapshots

dreamer-89 pushed a commit that referenced this pull request Aug 5, 2022
@opensearch-trigger-bot @reta
OpenSearch crashes on closed client connection before search reply wh…
ce9b988 
…en total ops higher compared to expected (#4143) (#4144)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
(cherry picked from commit 203f44e)

Co-authored-by: Andriy Redko <andriy.redko@aiven.io>
dreamer-89 pushed a commit that referenced this pull request Aug 5, 2022
@reta
OpenSearch crashes on closed client connection before search reply wh…
fe2aa61 
…en total ops higher compared to expected (#4143) (#4145)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
dreamer-89 pushed a commit that referenced this pull request Aug 5, 2022
@reta
OpenSearch crashes on closed client connection before search reply wh…
813a75a 
…en total ops higher compared to expected (#4143) (#4146)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
dreamer-89 pushed a commit that referenced this pull request Aug 5, 2022
@reta
OpenSearch crashes on closed client connection before search reply wh…
10280ef 
…en total ops higher compared to expected (#4143) (#4147)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
dreamer-89 pushed a commit that referenced this pull request Aug 5, 2022
@reta
OpenSearch crashes on closed client connection before search reply wh…
9b03eac 
…en total ops higher compared to expected (#4143) (#4148)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
dreamer-89 pushed a commit that referenced this pull request Aug 5, 2022
@reta
OpenSearch crashes on closed client connection before search reply wh…
4b0f2ec 
…en total ops higher compared to expected (#4143) (#4149)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
@dbonf
Copy link

dbonf commented Aug 6, 2022 •
edited
Loading

@reta I managed to build from reta:backport/backport-4143-to-1.3 and the tests are promising, no crashes

dreamer-89 pushed a commit to dreamer-89/OpenSearch that referenced this pull request Aug 12, 2022
@reta @dreamer-89
OpenSearch crashes on closed client connection before search reply wh…
86f5a04 
…en total ops higher compared to expected (opensearch-project#4143)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
@dreamer-89 dreamer-89 added the v2.3.0 'Issues and PRs related to version v2.3.0' label Sep 8, 2022
@kkewwei kkewwei mentioned this pull request Feb 4, 2024
Merged
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrross andrross andrross approved these changes

Assignees
No one assigned
Labels
backport 2.x Backport to 2.x branch v2.3.0 'Issues and PRs related to version v2.3.0'
Projects
None yet
Milestone
No milestone
5 participants
@reta @codecov-commenter @dbonf @andrross @dreamer-89